Here's what happened in the Reproducible Builds effort between Sunday October 1 and Saturday October 7 2017: Media coverage

• Bernhard sent another report about the status of Reproducible openSUSE. They currently they are at 478 unreproducible and 11,111 reproducible packages out of 11,821, so also at 93%!
• Holger attempted to get a Reproducible Builds devroom at FOSDEM 2018 but sadly this proposal was not accepted.

Documentation updates

• Christoph Berg created a wiki page about Openjade generated timestamps from DSSSL stylesheets.

Packages reviewed and fixed, and bugs filed

• Bernhard M. Wiedemann:
  • LiE uninitialized memory (need to find upstream)
  • chrony date (merged)
• Chris Lamb:
  • #877375 filed against polygen.
  • #877381 filed against plr.
  • #877384 filed against rcs.
  • #877928 filed against cadvisor.
• jathan:
  • #877470 filed against bsh.

Reviews of unreproducible packages 32 package reviews have been added, 46 have been updated and 62 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (27)

diffoscope development

• Chris Lamb:
  • Don't crash on malformed md5sums files. (Closes: #877473)
  • Improve names in output of "internal" binwalk members. (Closes: #877525)
• Mattia Rizzolo:
  • Fix test compatibility with dtb version 1.4.5

strip-nondeterminism development Rob Browning noticed that strip-nondeterminism was causing serious performance regressions in the Clojure programming language within Debian. After some discussion, Chris Lamb also posted a query to debian-devel in case there were any other programming languages that might be suffering from the same problem.

• Chris Lamb:
  • jar.pm: Clojure considers the .class file to be stale if it shares the same timestamp of the .clj. We thus adjust the timestamps of the .clj to always be younger.. (Closes: #877418)
  • jar.pm, zip.pm: Allow $options member_normalizer callback to support specifying the timestamp.
  • zip.pm: Ensure that we don't try and write an old timestamp; Archive::Zip will do this anyway, just noisily.
  • zip.pm: Calculate the target canonical time in just one place.
  • bin/strip-nondeterminism: Print a warning in --verbose mode if no canonical time specified.
  • jar.pm: Update comment to reflect that NTFS/FAT has a 2s timestamp granularity.
  • jar.pm: s/NTFS/FAT/. Thanks to James Ross.

reprotest development Versions 0.7.1 and 0.7.2 were uploaded to unstable by Ximin Luo:

• New features:
  • Add a --auto-build option to try to determine which specific variations cause unreproducibility.
  • Add a --source-pattern option to restrict copying of source_root, and set this automatically in our presets.
• Usability improvements:
  • Improve error messages in some common scenarios.
    • Fiving a source_root or build_command that doesn't exist
    • Using reprotest with default settings after not installing Recommends
  • Output hashes after a successful --auto-build.
  • Print a warning message if we reproduced successfully but didn't vary everything.
• Fix varying both umask and user_group at the same time.
• Have dpkg-source extract to different build dir if varying the build-path.
• Pass --exclude-directory-metadata to diffoscope(1) by default as this is the majority use-case.
• Various bug fixes to get the basic dsc+schroot example working.

It included contributions already covered by posts of the previous weeks, as well as new ones from:

• Ximin Luo:
  • main: Add a --env-build option for testing different env vars
  • Don't output spurious warnings in tests
  • Add some more notes on the remaining variations
  • Fix the help text for virtual servers

tests.reproducible-builds.org

• Mattia Rizzolo:
  • Re-deploy odxu4a after being reinstalled and renamed from odxu4.
• Vagrant Cascadian:
  • Rename armhf host odxu4 to odxu4a

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday September 24 and Saturday September 30 2017: Development and fixes in key packages Kai Harries did an initial packaging of the Nix package manager for Debian. You can track his progress in #877019. Uploads in Debian:

• Chris Lamb:
  • shadow/1:4.5-1 fixing #857803.
• Holger Levsen:
  • font-uralic/0.0.20040829-6 fixing #854362. (NMU)
  • mpack/1.6-8.2 fixing #777376. (NMU)

Packages reviewed and fixed, and bugs filed Patches sent upstream:

• Bernhard M. Wiedemann:
  • python-numpy build timestamp; merged
  • python-marshmallow build timestamp; merged
  • python-astropy-helpers build timestamp; merged
  • oprofile build timestamp
  • libheimdal build timestamp, hostname, username; upstream exploring alternative fixes
  • openSUSE/Qt hash table seed
  • libkolabxml/xsd ASLR memory location differences; no patch

Reproducible bugs (with patches) filed in Debian:

• Chris Lamb:
  • #877375 filed against polygen.
  • #877381 filed against plr.
  • #877384 filed against rcs.
• Daniel Schepler:
  • #876672 filed against e2fsprogs.
• Vagrant Cascadian:
  • #876657 filed against device-tree-compiler and uploaded 1.4.4-1 with the patch applied.

QA bugs filed in Debian:

• Adrian Bunk:
  • #876641 filed against pcb.
  • #876685 filed against mssh.
  • #876776 filed against fityk.
  • #876845 filed against webkit2-sharp.
  • #876870 filed against apertium-en-es.
  • #877021 filed against breathe.
  • #877031 filed against sextractor.
  • #877054 filed against hypre.
  • #877063 filed against libitpp.
  • #877065 filed against alglib.
  • #877211 filed against ipmiutil.

Reviews of unreproducible packages 103 package reviews have been added, 153 have been updated and 78 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (177)
• Andreas Beckmann (2)
• Daniel Schepler (1)

diffoscope development Mattia Rizzolo uploaded version 87 to stretch-backports.

• Holger Levsen:
  • Bump standards version to 4.1.1, no changes needed.

strip-nondeterminism development

• Holger Levsen:
  • Bump Standards-Version to 4.1.1, no changes needed.

reprotest development

• Ximin Luo:
  • New features:
    • Add a --env-build option for testing different env vars. (In-progress, requires the python-rstr package awaiting entry into Debian.)
    • Add a --source-pattern option to restrict copying of source_root.
  • Usability improvements:
    • Improve error messages in some common scenarios.
    • Output hashes after a successful --auto-build.
    • Print a warning message if we reproduced successfully but didn't vary everything.
    • Update examples in documentation.
  • Have dpkg-source extract to different build dir iff varying the build-path.
  • Pass --debug to diffoscope if verbosity >= 2.
  • Pass --exclude-directory-metadata to diffoscope(1) by default.
  • Much refactoring to support the other work and several minor bug fixes.
• Holger Levsen:
  • Bump standards version to 4.1.1, no changes needed.

tests.reproducible-builds.org

• Holger Levsen:
  • Fix scheduler to not send empty scheduling notifications in the rare cases nothing has been scheduled.
  • Fix colors in 'amount of packages build each day on $ARCH' graphs.

reproducible-website development

• Holger Levsen:
  • Fix up HTML syntax
  • Announce that RWS3 will happen at Betahaus, Berlin

Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann, Holger Levsen and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday September 17th and Saturday September 23rd 2017: Media coverage

• Christos Zoulas gave a talk entitled Reproducible builds on NetBSD at EuroBSDCon 2017

Reproducible work in other packages

• Paul Eggert reported that TZ=UTC is not a portable setting for the TZ environment variable.

Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #876615 filed against librsvg.
• Bernhard M. Wiedemann:
  • varnish (random IDs)
  • make (sort)
  • nautilus-dropbox (extended date)
  • fontforge (date)
  • votca-csg (merged, date)
  • freeipmi (merged, date)
  • libmypaint (merged, sort)
  • doomsday (merged, sort)
  • asciidoc (help make release with SOURCE_DATE_EPOCH patch)

Reviews of unreproducible packages 1 package reviews was added, 49 have been updated and 54 have been removed in this week, adding to our knowledge about identified issues. One issue type was updated:

• gtk_doc_api_index_full (fixed upstream: #779090)

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (56)
• Bas Couwenberg (1)
• Helmut Grohne (1)
• Nobuhiro Iwamatsu (2)

diffoscope development Version 87 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Ximin Luo:
  • comparators/*:
    • Add a test for fallback_recognizes and improve the behaviour
    • Add a fallback_recognizes to work around file(1) #876316. (Closes: #875272)
    • If --force-details then don't skip files with identical md5sums either
    • Add a --force-details flag for debugging
  • presenters.html:
    • Restore the previous more-detailed comment
    • Don't show pointer-cursor when jQuery is disabled
    • Prune all descendants properly (Closes: #875281)
  • difference.py:
    • Also copy self._comment properly, compare self._visuals in equals()
    • In fmap/map_lines, don't forget about self._visuals
    • Use diff_split_lines everywhere
  • tests:
    • Add case for #875281
    • Make test_md5sums less brittle
    • Update test_deb for new string
  • readers: Convert bytes to str in the right place
  • config: Force-set a value if it must be < another and it was not set on purpose (Closes: #875451)
  • Bump minimum Python version to 3.5 as we use syntax introduced by PEP 448
• Chris Lamb:
  • Print a debugging message if we are reading diff from stdin.
  • Compare types with identity not equality.
  • diffoscope.presenters.html: Use logging.py's lazy argument interpolation.
  • debian/control: Bump Standards-Version to 4.1.0.
• Mattia Rizzolo:
  • debian/rules: Place "before" commands before "after" commands.

strip-nondeterminism development

• Chris Lamb:
  • Log which handler procesed a file. (Closes: #876140)
  • Bump Standards-Version to 4.1.0.

reprotest development Version 0.7 was uploaded to unstable by Ximin Luo:

• Ximin Luo:
  • Push use of UNIX return codes to the edges of the program
  • Add a --auto-build option to determine which variations cause unreproducibility
  • Allow umask and user_group to both vary at the same time
  • Generate build names in main instead of build, guard against dupes
  • Pull traceback-printing stuff out of the core code
  • More refactoring, make check() contain only logic that would be changed in an auto-detector
  • Split check() into a coroutine producer and consumer, prepares auto-detection

tests.reproducible-builds.org Vagrant Cascadian and Holger Levsen:

• Re-add and armhf build node that had been disabled due to performance issues, but works linux 4.14-rc1 now! #876212

Holger Levsen:

• Use botch from stretch to fix the jenkins job which create the package sets. (botch is currently uninstallable in sid and from pre-stretch-release times we used a sid schroot to install and use botch.)

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Lintian is a static analysis tool for Debian packages, reporting on various errors, omissions and general quality-assurance issues to maintainers. I've previously written about my exploits with Lintian as well as authoring a short tutorial on how to write your own Lintian check. Anyway, I recently uploaded version 2.5.53 about two months since previous release. The biggest changes you may notice are supporting the latest version of the Debian Policy as well the addition of checks to encourage the migration to Python 3. Thanks to all who contributed patches, code review and bug reports to this release. The full changelog is as follows:

lintian (2.5.53) unstable; urgency=medium
  The "we are all Perl developers now" release.
  * Summary of tag changes:
    + Added:
      - alternatively-build-depends-on-python-sphinx-and-python3-sphinx
      - build-depends-on-python-sphinx-only
      - dependency-on-python-version-marked-for-end-of-life
      - maintainer-script-interpreter
      - missing-call-to-dpkg-maintscript-helper
      - node-package-install-in-nodejs-rootdir
      - override-file-in-wrong-package
      - package-installs-java-bytecode
      - python-foo-but-no-python3-foo
      - script-needs-depends-on-sensible-utils
      - script-uses-deprecated-nodejs-location
      - transitional-package-should-be-oldlibs-optional
      - unnecessary-testsuite-autopkgtest-header
      - vcs-browser-links-to-empty-view
    + Removed:
      - debug-package-should-be-priority-extra
      - missing-classpath
      - transitional-package-should-be-oldlibs-extra
  * checks/apache2.pm:
    + [CL] Fix an apache2-unparsable-dependency false positive by allowing
      periods (".") in dependency names.  (Closes: #873701)
  * checks/binaries.pm:
    + [CL] Apply patches from Guillem Jover & Boud Roukema to improve the
      description of the binary-file-built-without-LFS-support tag.
      (Closes: #874078)
  * checks/changes. pm,desc :
    + [CL] Ignore DFSG-repacked packages when checking for upstream
      source tarball signatures as they will never match by definition.
      (Closes: #871957)
    + [CL] Downgrade severity of orig-tarball-missing-upstream-signature
      from "E:" to "W:" as many common tools do not make including the
      signatures easy enough right now.  (Closes: #870722, #870069)
    + [CL] Expand the explanation of the
      orig-tarball-missing-upstream-signature tag to include the location
      of where dpkg-source will look. Thanks to Theodore Ts'o for the
      suggestion.
  * checks/copyright-file.pm:
    + [CL] Address a number of issues in copyright-year-in-future:
      - Prevent false positives in port numbers, email addresses, ISO
        standard numbers and matching specific and general street
        addresses.  (Closes: #869788)
      - Match all violating years in a line, not just the first (eg.
        "2000-2107").
      - Ignore meta copyright statements such as "Original Author". Thanks
        to Thorsten Alteholz for the bug report.  (Closes: #873323)
      - Expand testsuite.
  * checks/cruft. pm,desc :
    + [CL] Downgrade severity of file-contains-fixme-placeholder
      tag from "important" (ie. "E:") to "wishlist" (ie. "I:").
      Thanks to Gregor Herrmann for the suggestion.
    + [CL] Apply patch from Alex Muntada (alexm) to use "substr" instead
      of "substring" in mentions-deprecated-usr-lib-perl5-directory's
      description.  (Closes: #871767)
    + [CL] Don't check copyright_hints file for FIXME placeholders.
      (Closes: #872843)
    + [CL] Don't match quoted "FIXME" variants as they are almost always
      deliberate. Thanks to Adrian Bunk for the report.  (Closes: #870199)
    + [CL] Avoid false positives in missing source checks for "CSS Browser
      Selector".  (Closes: #874381)
  * checks/debhelper.pm:
    + [CL] Prevent a false positive of
      missing-build-dependency-for-dh_-command that can be exposed by
      following the advice for the recently added
      useless-autoreconf-build-depends tag.  (Closes: #869541)
  * checks/debian-readme. pm,desc :
    + [CL] Ensure readme-debian-contains-debmake-template also checks
      for templates "Automatically generated by debmake".
  * checks/description. desc,pm :
    + [CL] Clarify explanation of description-starts-with-leading-spaces
      tag. Thanks to Taylor Kline  for the report
      and patch.  (Closes: #849622)
    + [NT] Skip capitalization-error-in-description-synopsis for
      auto-generated packages (such as dbgsym packages).
  * checks/fields. desc,pm :
    + [CL] Ensure that python3-foo packages have "Section: python", not
      just python2-foo.  (Closes: #870272)
    + [RG] Do no longer require debug packages to be priority extra.
    + [BR] Use Lintian::Data for name/section mapping
    + [CL] Check for packages including "?rev=0&sc=0" in Vcs-Browser.
      (Closes: #681713)
    + [NT] Transitional packages should now be "oldlibs/optional" rather
      than "oldlibs/extra".  The related tag has been renamed accordingly.
  * checks/filename-length.pm:
    + [NT] Skip the check on auto-generated binary packages (such as
      dbgsym packages).
  * checks/files. pm,desc :
    + [BR] Avoid privacy-breach-generic false positives for legal.xml.
    + [BR] Detect install of node package under /usr/lib/nodejs/[^/]*$
    + [CL] Check for packages shipping compiled Java class files. Thanks
      Carn  Draug .  (Closes: #873211)
    + [BR] Privacy breach is no longer experimental.
  * checks/init.d.desc:
    + [RG] Do not recommend a versioned dependency on lsb-base in
      init.d-script-needs-depends-on-lsb-base.  (Closes: #847144)
  * checks/java.pm:
    + [CL] Additionally consider .cljc files as code to avoid false-
      positive codeless-jar warnings.  (Closes: #870649)
    + [CL] Drop problematic missing-classpath check.  (Closes: #857123)
  * checks/menu-format.desc:
    + [CL] Prevent false positives in desktop-entry-lacks-keywords-entry
      for "Link" and "Directory" .desktop files.  (Closes: #873702)
  * checks/python. pm,desc :
    + [CL] Split out Python checks from "scripts" check to a new, source
      check of type "source".
    + [CL] Check for python-foo without corresponding python3-foo packages
      to assist in Python 2.x deprecation.  (Closes: #870681)
    + [CL] Check for packages that Build-Depend on python-sphinx only.
      (Closes: #870730)
    + [CL] Check for packages that alternatively Build-Depend on the
      Python 2 and Python 3 versions of Sphinx.  (Closes: #870758)
    + [CL] Check for binary packages that depend on Python 2.x.
      (Closes: #870822)
  * checks/scripts.pm:
    + [CL] Correct false positives in
      unconditional-use-of-dpkg-statoverride by detecting "if !" as a
      valid shell prefix.  (Closes: #869587)
    + [CL] Check for missing calls to dpkg-maintscript-helper(1) in
      maintainer scripts.  (Closes: #872042)
    + [CL] Check for packages using sensible-utils without declaring a
      dependency after its split from debianutils.  (Closes: #872611)
    + [CL] Warn about scripts using "nodejs" as an interpreter now that
      nodejs provides /usr/bin/node.  (Closes: #873096)
    + [BR] Add a statistic tag giving interpreter.
  * checks/testsuite. desc,pm :
    + [CL] Remove recommendations to add a "Testsuite: autopkgtest" field
      to debian/control as it is added when needed by dpkg-source(1)
      since dpkg 1.17.1.  (Closes: #865531)
    + [CL] Warn if we see an unnecessary "Testsuite: autopkgtest" header
      in debian/control.
    + [NT] Recognise "autopkgtest-pkg-go" as a valid test suite.
    + [CL] Recognise "autopkgtest-pkg-elpa" as a valid test suite.
      (Closes: #873458)
    + [CL] Recognise "autopkgtest-pkg-octave" as a valid test suite.
      (Closes: #875985)
    + [CL] Update the description of unknown-testsuite to reflect that
      "autopkgtest" is not the only valid value; the referenced URL
      is out-of-date (filed as #876008).  (Closes: #876003)
  * data/binaries/embedded-libs:
    + [RG] Detect embedded copies of heimdal, libgxps, libquicktime,
      libsass, libytnef, and taglib.
    + [RG] Use an additional string to detect embedded copies of
      openjpeg2.  (Closes: #762956)
  * data/fields/name_section_mappings:
    + [BR] node- package section is javascript.
    + [CL] Apply patch from Guillem Jover to add more section mappings.
      (Closes: #874121)
  * data/fields/obsolete-packages:
    + [MR] Add dh-systemd.  (Closes: #872076)
  * data/fields/perl-provides:
    + [CL] Refresh perl provides.
  * data/fields/virtual-packages:
    + [CL] Update data file from archive. This fixes a false positive for
      "bacula-director".  (Closes: #835120)
  * data/files/obsolete-paths:
    + [CL] Add note to /etc/bash_completion.d entry regarding stricter
      filename requirements.  (Closes: #814599)
  * data/files/privacy-breaker-websites:
    + [BR] Detect custom donation logos like apache.
    + [BR] Detect generic counter website.
  * data/standards-version/release-dates:
    + [CL] Add 4.0.1 and 4.1.0 as known standards versions.
      (Closes: #875509)
  * debian/control:
    + [CL] Mention Debian Policy v4.1.0 in the description.
    + [CL] Add myself to Uploaders.
    + [CL] Drop unnecessary "Testsuite: autopkgtest"; this is implied from
      debian/tests/control existing.
  * commands/info.pm:
    + [CL] Add a --list-tags option to print all tags Lintian knows about.
      Thanks to Rajendra Gokhale for the suggestion.  (Closes: #779675)
  * commands/lintian.pm:
    + [CL] Apply patch from Maia Everett to avoid British spelling when
      using en_US locale.  (Closes: #868897)
  * lib/Lintian/Check.pm:
    + [CL] Stop emitting  maintainer,uploader -address-causes-mail-loops
      for @packages.debian.org addresses.  (Closes: #871575)
  * lib/Lintian/Collect/Binary.pm:
    + [NT] Introduce an "auto-generated" argument for "is_pkg_class".
  * lib/Lintian/Data.pm:
    + [CL] Modify Lintian::Data's "all" to always return keys in insertion
      order, dropping dependency on libtie-ixhash-perl.
  * helpers/coll/objdump-info-helper:
    + [CL] Apply patch from Steve Langasek to accommodate binutils 2.29
      outputting symbols in a different format on ppc64el.
      (Closes: #869750)
  * t/tests/fields-perl-provides/tags:
    + [CL] Update expected output to match new Perl provides.
  * t/tests/files-privacybreach/*:
    + [CL] Add explicit test for packages including external fonts via
      the Google Font API. Thanks to Ian Jackson for the report.
      (Closes: #873434)
    + [CL] Add explicit test for packages including external fonts via
      the Typekit API via <script/> HTML tags.
  * t/tests/*/desc:
    + [CL] Add missing entries in "Test-For" fields to make
      development/testing workflow less error-prone.
  * private/generate-tag-summary:
    + [CL] git-describe(1) will usually emit 7 hexadecimal digits as the
      abbreviated object name,  However, as this can be user-dependent,
      pass --abbrev=0 to ensure it does not vary between systems.  This
      also means we do not need to strip it ourselves.
  * private/refresh-*:
    + [CL] Use deb.debian.org as the default mirror.
    + [CL] Update locations of Contents-<arch> files; they are now
      namespaced by distribution (eg. "main").
 -- Chris Lamb <lamby@debian.org>  Wed, 20 Sep 2017 09:25:06 +0100

Here's what happened in the Reproducible Builds effort between Sunday September 3 and Saturday September 9 2017: Media coverage

• isdebianreproducibleyet.com was released and subsequently updated.

GSoC and Outreachy updates Debian will participate in this year's Outreachy initiative and the Reproducible Builds is soliciting mentors and students to join this round. For more background please see the following mailing list posts: 1, 2 & 3. Reproduciblity work in Debian

• Chris Lamb filed #874102 filed against texlive-bin to incorporate a proposed upstream change to fix reproducibility issues in generated PDF files.

In addition, the following NMUs were accepted:

• fastforward (#776972) (lamby)
• dtc-xen (#777322) (lamby)
• dhcpping (#777320) (lamby)
• vimoutliner (#776369) (lamby)

Reproduciblity work in other projects

• The Linux kernel announced support for the randstruct GCC plugin.
• "Please make the output of gio-querymodules deterministic" was merged upstream. (lamby)

Patches sent upstream:

• Bernhard M. Wiedemann:
  • gcin: (merged) Uninitialized stack memory
  • html5-parser (merged): Sorting
  • gromacs (merged): Date
  • crawl (merged): Date
  • GCompris-gtk: Date
  • heimdal: Date, hostname
• Chris Lamb:
  • Numpy (#872459) (PR #9652) build path (merged)

Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #874186 filed against svgpp.

Reviews of unreproducible packages 3 package reviews have been added, 2 have been updated and 2 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (15)

diffoscope development Development continued in git, including the following contributions:

• Chris Lamb:
  • Add support for "binwalking" to find (eg.) concatenated CPIO archives. (Closes: #820631)
  • Loosen matching of file(1)'s output to ensure we correctly also match TTF files under file 5.32.
  • Check we identify all CPIO fixtures in tests
  • Make failing a some flake8 tests cause the testsuite to fail. (Currently just "undefined name")
  • Countless style fixups, eg. remove unused imports, removing blank lines from end of flies, etc.
  • Compare None using identity, not equality.
  • diffoscope.diff: Correct reference to self.buf.
  • comparators.utils.file: Correct reference to path_apparent_size.
• Juliana Rodrigues:
  • Sip html_visuals test if 'sng' binary is not available
• Mattia Rizzolo:
  • Numerous PEP8 fixes (eg. E122, E302, E713, etc.)

Mattia Rizzolo also uploaded the version 86 released last week to stretch-backports. reprotest development

• Santiago Torres:
  • Correct string formatting in get_all_servers
• Ximin Luo:
  • Heavy refactoring towards supporting running more than 2 builds
  • Rename append_command to append_to_build_command

tests.reproducible-builds.org

• h01ger:
  • Don't update the stretch package sets anymore
  • Update URL for Tails packages list
  • Disabled the OpenWrt at the request of lynxis as they were broken; if no-one shows up to fix them, we'll probably remove them in the future, as all current development happens within LEDE.
  • Renewed the Let's Encrypt SSL certificates.
• Mattia Rizzolo:
  • Avoid stale temporary files from reproducible_build.sh
  • Avoid removing the temporary directory twice
  • Fix IRC notifications when somebody asks for artifacts
  • Simplify 'if' conditions

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday August 27 and Saturday September 2 2017: Talks and presentations Holger Levsen talked about our progress and our still-far goals at BornHack 2017 (Video). Toolchain development and fixes The Debian FTP archive will now reject changelogs where different entries have the same timestamps. UDD now uses reproducible-tracker.json (~25MB) which ignores our tests for Debian unstable, instead of our full set of results in reproducible.json. Our tests for Debian unstable uses a stricter definition of "reproducible" than what was recently added to Debian policy, and these stricter tests are currently more unreliable. Packages reviewed and fixed, and bugs filed Patches sent upstream:

• Bernhard M. Wiedemann:
  • File ordering:
    • klee-uclibc: sort
    • libdnet: sort
    • libinvm-cim: sort
    • libinvm-cli: sort
  • Embedded build-date timestamps:
    • robinhood: SOURCE_DATE_EPOCH support
    • ceph/rocksdb: SOURCE_DATE_EPOCH support
    • hylafax: use changelog modtime
    • gnucash: use changelog modtime
  • Warzone2100, merged: omit timestamps, sort file lists
• Chris Lamb:
  • glib2.0: sort file lists
  • (old) xorg, merged: SOURCE_DATE_EPOCH support

Debian bugs filed:

• Adrian Bunk:
  • #873608 filed against uhd.
  • #874186 filed against svgpp.
• Chris Lamb:
  • #873625 filed against glib2.0, filed upstream.
  • #874102 filed against texlive-bin.

Debian packages NMU-uploaded:

• Chris Lamb:
  • bittornado/0.3.18-10.3 from #796212
  • cgilib/0.6-1.1 from #776935
  • dict-gazetteer2k/1.0.0-5.4 from #776376
  • dict-moby-thesaurus/1.0-6.4 from #776375
  • dtaus/0.9-1.1 from #777321
  • wily/0.13.41-7.3 from #777360

Reviews of unreproducible packages 25 package reviews have been added, 50 have been updated and 86 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (46)
• Mart n Ferrari (1)
• Steve Langasek (1)

diffoscope development Version 86 was uploaded to unstable by Mattia Rizzolo. It included previous weeks' contributions from:

• Mattia Rizzolo
  • tests/binary: skip a test if the 'distro' module is not available.
  • Some code quality and style improvements.
• Guangyuan Yang
  • tests/iso9660: support both cdrtools' genisoimage's versions of isoinfo.
• Chris Lamb
  • comparators/xml: Use name attribute over path to avoid leaking comparison full path in output.
  • Tidy diffoscope.progress a little.
• Ximin Luo
  • Add a --tool-prefix-binutils CLI flag. Closes: #869868
  • On non-GNU systems, prefer some tools that start with "g". Closes: #871029
  • presenters/html: Don't traverse children whose parents were already limited. Closes: #871413
• Santiago Torres-Arias
  • diffoscope.progress: Support the new fork of python-progressbar. Closes: #873157

reprotest development Development continued in git with contributions from:

• Ximin Luo:
  • Add -v/--verbose which is a bit more popular.
  • Make it possible to omit "auto" when building packages.
  • Refactor how the config file works, in preparation for new features.
  • chown -h for security.

Misc. This week's edition was written by Ximin Luo, Chris Lamb, Bernhard M. Wiedemann and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday August 20 and Saturday August 26 2017: Debian development

• "Packages should build reproducibly" was released in Debian Policy 4.1.0.0. For more background please see last week's post.
• A patch by Chris Lamb to make Dpkg::Substvars warnings output deterministic was merged by Guillem Jover. This helps the Reproducible Builds effort as it removes unnecessary differences in logs of two package builds. (#870221)

Packages reviewed and fixed, and bugs filed Forwarded upstream:

• Debian bug #872728 filed against desktop-file-utils. Filed upstream. (lamby, via a reproducibility issue in Tails).
• Debian bug #872729 filed against gtk+2.0. Filed upstream. (lamby, via a reproducibility issue in Tails).
• sawfish:
  • SOURCE_DATE_EPOCH
  • .tar.gz (merged)
  • Sorting (merged)
• librep:
  • SOURCE_DATE_EPOCH (merged)
  • Sorting (merged)
• zstd (merged) sort
• lirc (solved)
• autogen (tar.gz)

Accepted repoducibility NMUs in Debian:

• Chris Lamb:
  • jsmath-fonts (for bug #792319).
  • xvier (for bug #777330).
• Mattia Rizzolo:
  • aewan (for bug #842864).

Other issues:

• Adrian Bunk:
  • #872675 filed against pdp.
  • #872678 filed against gem.
  • #872860 filed against csound.
  • #872918 filed against sketch.
• Dmitry Shachnev:
  • #872992 filed against pytest-qt.

Reviews of unreproducible packages 16 package reviews have been added, 38 have been updated and 48 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated:

• Add new captures_build_dir_in_qmake_prl_files toolchain issue (Chris Lamb)
• Remove ftbfs_uninvestigated_big_packages (Mattia Rizzolo)

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (37)
• Dmitry Shachnev (1)
• James Cowgill (1)

diffoscope development

• Chris Lamb:
  • Tidy diffoscope.progress a little.
• Santiago Torres-Arias:
  • diffoscope.progress: Support the new fork of python-progressbar. (#873157)

disorderfs development Version 0.5.2-1 was uploaded to unstable by Ximin Luo. It included contributions from:

• Ximin Luo:
  • Add -q,--quiet CLI option
  • Update to latest Standards-Version; no changes required
  • Update debian/changelog
  • Update NEWS
  • Adopt as team maintenance
  • Add my signing key only
  • Add instructions for Debian release
• Holger Levsen:
  • Add myself to uploaders
  • Move package to 'optional' section

reprotest development

• Ximin Luo:
  • Don't vary user_group in the tests - it's not usually possible in automated builders
  • Default verbosity to 0 and silence disorderfs if set
  • Add the ability to vary the user. (#872412)
  • Add TODO

Misc. This week's edition was written in alphabetical order by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday August 13 and Saturday August 19 2017: Reproducible Builds finally mandated by Debian Policy "Packages should build reproducibly" was merged into Debian policy! The added text is as follows and has been included into debian-policy 4.1.0.0:

Reproducibility
---------------
Packages should build reproducibly, which for the purposes of this
document [#]_ means that given
- a version of a source package unpacked at a given path;
- a set of versions of installed build dependencies;
- a set of environment variable values;
- a build architecture; and
- a host architecture,
repeatedly building the source package for the build architecture on
any machine of the host architecture with those versions of the build
dependencies installed and exactly those environment variable values
set will produce bit-for-bit identical binary packages.
It is recommended that packages produce bit-for-bit identical binaries
even if most environment variables and build paths are varied.  It is
intended for this stricter standard to replace the above when it is
easier for packages to meet it.
.. [#]
   This is Debian's precisification of the  reproducible-builds.org
   definition  _.

• Holger Levsen wrote a blog post briefly describing the background and implications of this. To quote him: "we are not 94% done yet, rather more like half done or so. We still need tools and processes to enable anyone to indepently verify that a given binary comes from the sources it is said to be coming, this will involve distributing .buildinfo files and providing user interfaces in APT and elsewhere and probably also systematic rebuilds by us and other parties. And 6% or 7% of the archive is still a lot of packages, eg. in Buster we currently still have 273 unreproducible key packages and for a large part we don't have patches yet so there is still a lot of work ahead."
• There were discussion threads on Hacker News and Reddit.
• Our long-term goal is that Policy mandates that packages "must" be reproducible, but for that we need to show further progress and also reach a consensus on .buildinfo files and much more.

Reproducible work in other projects Bernhard M. Wiedemann's reproducibleopensuse scripts now work on Debian buster on the openSUSE Build Service with the latest versions of osc and obs-build. Toolchain development and fixes #872514 was opened on devscripts by Chris Lamb to add a reproducible-check program to report on the reproducibility status of installed packages. Packages reviewed and fixed, and bugs filed Upstream reports:

• Bernhard M. Wiedemann:
  • qt5/base, SOURCE_DATE_EPOCH support.

Debian reports:

• Adrian Bunk:
  • #872262 filed against jellyfish.
  • #872675 filed against pdp.
  • #872678 filed against gem.
• Chris Lamb:
  • #872453 filed against isa-support.
  • #872459 filed against python-numpy.
  • #872460 filed against gcab, forwarded upstream.
  • #872514 filed against devscripts.
  • #872728 filed against desktop-file-utils.
  • #872729 filed against gtk+2.0, forwarded upstream, found via a reproducibility issue in Tails.
• Federico Brega:
  • #872285 filed against pyqt5.
• Jeremy Bicha:
  • #871963 filed against birdfont.
• Philip Rinn:
  • #872184 filed against quilt.

Debian non-maintainer uploads:

• Mattia Rizzolo:
  • console-data (for bug #799871).

Reviews of unreproducible packages 47 package reviews have been added, 58 have been updated and 39 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been updated:

• Added nondeterministic_output_generated_by_gcab.
• Added build_path_captured_by_python_numpy_misc_util.
• Added captures_build_path_in_sphinx_attr_links.
• Re-added nondeterminstic_ordering_in_gsettings_glib_enums_xml.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (59)
• Bastien Roucari s (1)
• James Clarke (1)
• Jeremy Bicha (1)

diffoscope development Development continued in git, including the following contributions:

• Ximin Luo:
  • presenters: html: Don't traverse children whose parents were already limited (Closes: #871413)
  • On a non-GNU system, prefer tools that start with "g" for certain whitelisted commands. (Closes: #871029)
  • Add a --tool-prefix-binutils CLI flag. (Closes: #869868)
• Chris Lamb:
  • Temporarily revert "Bump Standards-Version to 4.0.1" to avoid spurious CI test failures.
  • comparators.xml: Use name attribute over path to avoid leaking comparison full path in output.
  • Code style fixes.

disorderfs development Development continued in git, including the following contributions:

• Chris Lamb:
  • Add simple autopkgtest.

reprotest development Development continued in git, including the following contributions:

• Ximin Luo:
  • Choose an existent HOME for the "control" build. (Closes: #860428)
  • Update debian/changelog with Santiago's changes.
• Santiago Torres:
  • Abstract parts of autopkgtest to support running on non-Debian systems.
  • Add a --host-distro flag to support that too.

tests.reproducible-builds.org Mattia fixed the script which creates the HTML representation of our database scheme to not append .html twice to the filename. Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday 6th and Saturday 12th August 2017:

• There were two talks mainly about Reproducible Builds at DebConf17:
  • Reproducible builds: Status update by Chris Lamb, Holger Levsen, Maria Glukhova, Steven Chamberlain, Vagrant Cascadian, Valerie Young and Ximin Luo.
  • Fun with .buildinfo by Steven Chamberlain.
• Reproducible builds were also mentioned in many other talks including:
  • Using Qubes OS from the POV of a Debian developer by Holger Levsen.
  • Debian for Medical Software by Andr Roth.
  • Let's maintain jenkins.debian.org as a team by Holger Levsen.
  • Signing package contents: why and how by Matthew Garrett.
  • Bits from the DPL by Chris Lamb.
  • Installing Debian by Vagrant Cascadian.
  • in-toto -- Securing supply chains as a whole by Lukas Puehringer.
  • Debian Cloud BoF by Steve McIntyre.
  • Will there be Debian in your next BMW car? by Aigars Mahinovs.
• There has also been substantial activity regarding updating the Debian Policy regarding Reproducible Builds (#844431). This work is still ongoing and will be covered in future issues of this weekly blog.

Notes about reviews of unreproducible packages 13 package reviews have been added, 7 have been updated and 34 have been removed in this week, adding to our knowledge about identified issues. Packages reviewed and fixed, and reproducibility related bugs filed

• Adrian Bunk:
  • #871818 filed against debian-zh-faq.
  • #871907 filed against praat.
• Katsuhiko Nishimra:
  • #871591 filed against llvm-toolchain-snapshot.
• Lucas Nussbaum:
  • #871059 filed against fltk1.3.
  • #871155 filed against brltty.
  • #871159 filed against texlive-extra.
  • #871345 filed against texlive-base.
  • #871349 filed against ispell-uk.
  • #871357 filed against po4a.
• jathan:
  • #870890 filed against apg.

Upstream packages:

• Bernhard M. Wiedemann:
  • enblend (merged), SOURCE_DATE_EPOCH support
  • systemtap
  • splint (dead project)
  • skelcd-openSUSE

Other bugs filed

• During our reproducibility testing, Adrian Bunk filed 48 FTBFS bugs this week.

diffoscope development

• Mattia Rizzolo:
  • debian/copyright: coalesce some file paragraphs and update information.
  • Bump Standards-Version to 4.0.1, no changes needed.

trydiffoscope development

• Chris Lamb:
  • Test --help output in autopkgtests

tests.reproducible-builds.org

• Mattia:
  • Notify the#debian-reproducible-changes IRC channel for unreproducible -> FTBFS transitions.
  • Update squid.conf for all nodes to 5.2.23 (and fixup some).
  • Enable the Munin Squid plugin on the Codethink arm64 nodes as well.
  • Force reconfiguration of Apache and Munin when update_jdn.sh is updated.
• Holger worked on slides for his DebConf17 BoF about migrating to jenkins.debian.org, which affects tests.r-b.o as well.

Misc. This week's edition was written by Chris Lamb & Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday July 30 and Saturday August 5 2017: Media coverage We were mentioned on Late Night Linux Episode 17, around 29:30. Packages reviewed and fixed, and bugs filed Upstream packages:

• Bernhard M. Wiedemann:
  • efl (merged), unique ids based on memory address
  • 389-ds (merged), SOURCE_DATE_EPOCH support.
  • plowshare, SOURCE_DATE_EPOCH support
  • sphinx, file ordering
  • sphinx, SOURCE_DATE_EPOCH support

Debian packages:

• Adrian Bunk:
  • #870212 filed against glib2.0.
  • #870213 filed against pajeng.
  • #870733 filed against openhpi.
  • #870738 filed against gtk2-engines-xfce.
  • #870741 filed against libgda5.
  • #870742 filed against libgnome.
  • #870749 filed against glade.
  • #870821 filed against esys-particle.
• Chris Lamb:
  • #870131 filed against autopep8.
  • #870221 filed against dpkg.
  • #870573 filed against grap.
• Johannes Schauer:
  • #870585 filed against hevea.
• Logan Rosen:
  • #870586 filed against behave.
• Lucas Nussbaum:
  • #871059 filed against fltk1.3.
  • #871155 filed against brltty.
  • #871159 filed against texlive-extra.
• jathan:
  • #870890 filed against apg.

Reviews of unreproducible packages 29 package reviews have been added, 72 have been updated and 151 have been removed in this week, adding to our knowledge about identified issues. 4 issue types have been updated:

• Added timestamps_generated_by_hevea.
• Added timestamps_in_source_generated_by_rcc.
• Updated build_id_differences_only: remove an obsolete example.
• Updated golang_compiler_captures_build_path_in_binary: mark as not deterministic, because the patch fixing it is not yet upstreamed.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (36)
• Andreas Beckmann (2)
• Daniel Schepler (2)
• Logan Rosen (1)
• Lucas Nussbaum (93)

diffoscope development Version 85 was uploaded to unstable by Mattia Rizzolo. It included contributions from:

• Mattia Rizzolo:
  • Add an explicit Recommends: on the defusedxml python package.
  • Various other code quality tweaks.
• Juliana Oliveira Rodrigues:
  • Fix test_ico_image for ImageMagick identify >= 6.9.8.
  • Use the defusedxml XML library by default in the XML comparator, if it's available. This protects against various XML parser DoS attacks and other security holes, which other Python XML libraries are vulnerable to.
• Ximin Luo:
  • Force a flush when writing output to diff. (Closes: #870049).

as well as previous weeks' contributions, summarised in the changelog. There were also further commits in git, which will be released in a later version:

• Guangyuan Yang:
  • tests/iso9660: support isoinfo's output coming from cdrtools' version instead of genisoimage's
• Mattia Rizzolo:
  • Code quality and test fixes.
• Chris Lamb:
  • Code quality and test fixes.

Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday July 23 and Saturday July 29 2017:

• Mattia posted an extensive status update from the Debian Reproducible Builds project to the debian-devel-announce mailing list. There were subsequent discussions on Hacker News and Reddit.
• A NASA presentation on "Real system failures" contained an interesting slide pertaining to reproducible builds.
• A 2015 article on Reproducible Builds ("How Debian Is Trying to Shut Down the CIA and Make Software Trustworthy Again") resurfaced in popularity, possibly due to the recent Wikileaks "Vault 7" exposure of malware whose effects can be ameliorated or detected by adopting Reproducible Builds.

Toolchain development and fixes

• Chris Lamb sent an experimental patch to apt to make the output of apt-ftparchive reproducible. Thanks to David Kalnischkies for reworking the result. (#869557)

Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #869578 filed against gdmap.
  • #869580 filed against teg.
  • #869583 filed against gnome-specimen.
  • #869884 filed against chemical-mime-data.
  • #870047 filed against imagemagick.
  • #870068 filed against kde4libs.
• Chris Lamb:
  • #869516 filed against libcdio.
  • A previous pull request against wheel was merged upstream.
• Helmut Grohne:
  • #869584 filed against fontconfig.
  • #869588 filed against libcap2.

Reviews of unreproducible packages 4 package reviews have been added, 2 have been updated and 24 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Aaron M. Ucko (1)
• Adrian Bunk (35)
• Helmut Grohne (4)
• Stefan Tatschner (1)

diffoscope development

• Juliana Oliveira Rodrigues:
  • Refactor XML comparator to use File.recognizes logic
  • Fix test_ico_image for identify >= 6.9.8
  • Fix test_no_android_manifest from test_apk
  • Fix test_android_manifest from test_apk
  • Fix test_xml_expected_diff output encoding to UTF-8
• Ximin Luo:
  • Improve method for detecting APKs
  • Factor common logic from various comparators into File.recognizes

Misc. This week's edition was written by Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday July 16 and Saturday July 22 2017: Toolchain development Bernhard M. Wiedemann wrote a tool to automatically run through different sources of non-determinism, and report which of these caused irreproducibility. Dan Kegel's patches to fpm were merged. Bugs filed Patches submitted upstream:

• Bernhard M. Wiedemann:
  • Sort file lists:
    • eric5 merged
    • libsass-python merged
    • tcl merged
    • python3.6 in progress
    • drbd
    • blobwars
  • Instead of fixing ordering issues in a custom .pak archive (same as blobwars above), we allow to install individual data files to avoid the issue:
    • edgar avoid sort
  • Omit the build date entirely:
    • fence-agents
  • SOURCE_DATE_EPOCH support:
    • criu, merged
    • dapl, merged
    • shorewall, merged
    • youtube-dl in progress
    • automake
    • crosstool-ng
    • docker
    • drbd
    • drbd-utils
    • getdp
    • infinipath-psm
    • opa-fm
    • opa-psm2
    • texinfo
  • geany/glfw unknown

Patches filed in Debian:

• Adrian Bunk:
  • #868599 filed against ocaml-curses.
  • #868609 filed against le.
  • #868612 filed against mixxx.
  • #868855 filed against softhsm2.
  • #868858 filed against gwc.
  • #869086 filed against dsniff.
• Chris Lamb:
  • #868790 filed against castle-game-engine, forwarded upstream.
  • #868843 filed against xorg-server, forwarded upstream.
  • #869516 filed against libcdio.
• Drew Parsons:
  • #868505 filed against sdpa.
• Lucas Nussbaum:
  • #868904 filed against gwc.
  • #868927 filed against python-pybedtools.
• Sascha Steinbiss:
  • #868772 filed against ragel.

Reviews of unreproducible packages 73 package reviews have been added, 44 have been updated and 50 have been removed in this week, adding to our knowledge about identified issues. No issue types were updated. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (106)
• Daniel Stender (1)
• Drew Parsons (1)
• F lix Sipma (1)
• Lucas Nussbaum (25)

diffoscope development

• Juliana Rodrigues:
  • Add new XML comparator. (Closes: #866120)
• Guangyuan Yang:
  • Fix 2 cases in test_device on FreeBSD
• Chris Lamb:
  • comparators.xml: Fix EPUB "missing file" tests; they ship a META-INF/container.xml file.
  • comparators.sqlite: Simplify file detection in Sqlite3Database.RE_FILE_TYPE
  • Style and attribution fixes to XML comparator and comparators.directory
• Ximin Luo:
  • main, logging: restore old logger settings to avoid pytest vomiting in certain situations
  • comparators/directory: Fix #868534 by expecting less strict test output

reprotest development

• Ximin Luo:
  • Use autopkgtest upstream paths, makes things easier to import
  • Add script for importing autopkgtest code, and import autopkgtest 4.4

Ximin also restarted the discussion with autopkgtest-devel about code reuse for reprotest. Santiago Torres began a series of patches to make reprotest more distro-agnostic, with the aim of making it usable on Arch Linux. Ximin reviewed these patches. Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Since debhelper/10.3, there has been a number of performance related changes. The vast majority primarily improves bulk performance or only have visible effects at larger input sizes. Most visible cases are:

• dh + dh_* now scales a lot better for large number of binary packages. Even more so with parallel builds.
• Most dh_* tools are now a lot faster when creating many directories or installing files.
• dh_prep and dh_clean now bulk removals.
• dh_install can now bulk some installations. For a concrete corner-case, libssl-doc went from approximately 11 seconds to less than a second. This optimization is implicitly disabled with exclude (among other).
• dh_installman now scales a lot better with many manpages. Even more so with parallel builds.
• dh_installman has restored its performance under fakeroot (regression since 10.2.2)

For debhelper, this mostly involved:

• avoiding fork+exec of commands for things doable natively in perl. Especially, when each fork+exec only process one file or dir.
• bulking as many files/dirs into the call as possible, where fork+exec is still used.
• caching / memorizing slow calls (e.g. in parts of pkgfile inside Dh_Lib)
• adding an internal API for dh to do bulk check for pkgfiles. This is useful for dh when checking if it should optimize out a helper.
• and, of course, doing things in parallel where trivially possible.

How to take advantage of these improvements in tools that use Dh_Lib:

• If you use install_ file,prog,lib,dir , then it will come out of the box. These functions are available in Debian/stable. On a related note, if you use doit to call install (or mkdir ), then please consider migrating to these functions instead.
• If you need to reset owner+mode (chown 0:0 FILE + chmod MODE FILE), consider using reset_perm_and_owner. This is also available in Debian/stable.
  • CAVEAT: It is not recursive and YMMV if you do not need the chown call (due to fakeroot).
• If you have a lot of items to be processed by a external tool, consider using xargs(). Since 10.5.1, it is now possible to insert the items anywhere in the command rather than just in the end.
• If you need to remove files, consider using the new rm_files function. It removes files and silently ignores if a file does not exist. It is also available since 10.5.1.
• If you need to create symlinks, please consider using make_symlink (available in Debian/stable) or make_symlink_raw_target (since 10.5.1). The former creates policy compliant symlinks (e.g. fixup absolute symlinks that should have been relative). The latter is closer to a ln -s call.
• If you need to rename a file, please consider using rename_path (since 10.5). It behaves mostly like mv -f but requires dest to be a (non-existing) file.
• Have a look at whether on_pkgs_in_parallel() / on_items_in_parallel() would be suitable for enabling parallelization in your tool.
  • The emphasis for these functions is on making parallelization easy to add with minimal code changes. It pre-distributes the items which can lead to unbalanced workloads, where some processes are idle while a few keeps working.

Credits: I would like to thank the following for reporting performance issues, regressions or/and providing patches. The list is in no particular order:

• Helmut Grohne
• Kurt Roeckx
• Gianfranco Costamagna
• Iain Lane
• Sven Joachim
• Adrian Bunk
• Michael Stapelberg

Should I have missed your contribution, please do not hesitate to let me know.
Filed under: Debhelper, Debian

Since debhelper/10.3, there has been a number of performance related changes. The vast majority primarily improves bulk performance or only have visible effects at larger input sizes. Most visible cases are:

• dh + dh_* now scales a lot better for large number of binary packages. Even more so with parallel builds.
• Most dh_* tools are now a lot faster when creating many directories or installing files.
• dh_prep and dh_clean now bulk removals.
• dh_install can now bulk some installations. For a concrete corner-case, libssl-doc went from approximately 11 seconds to less than a second. This optimization is implicitly disabled with exclude (among other).
• dh_installman now scales a lot better with many manpages. Even more so with parallel builds.
• dh_installman has restored its performance under fakeroot (regression since 10.2.2)

For debhelper, this mostly involved:

• avoiding fork+exec of commands for things doable natively in perl. Especially, when each fork+exec only process one file or dir.
• bulking as many files/dirs into the call as possible, where fork+exec is still used.
• caching / memorizing slow calls (e.g. in parts of pkgfile inside Dh_Lib)
• adding an internal API for dh to do bulk check for pkgfiles. This is useful for dh when checking if it should optimize out a helper.
• and, of course, doing things in parallel where trivially possible.

How to take advantage of these improvements in tools that use Dh_Lib:

• If you use install_ file,prog,lib,dir , then it will come out of the box. These functions are available in Debian/stable. On a related note, if you use doit to call install (or mkdir ), then please consider migrating to these functions instead.
• If you need to reset owner+mode (chown 0:0 FILE + chmod MODE FILE), consider using reset_perm_and_owner. This is also available in Debian/stable.
  • CAVEAT: It is not recursive and YMMV if you do not need the chown call (due to fakeroot).
• If you have a lot of items to be processed by a external tool, consider using xargs(). Since 10.5.1, it is now possible to insert the items anywhere in the command rather than just in the end.
• If you need to remove files, consider using the new rm_files function. It removes files and silently ignores if a file does not exist. It is also available since 10.5.1.
• If you need to create symlinks, please consider using make_symlink (available in Debian/stable) or make_symlink_raw_target (since 10.5.1). The former creates policy compliant symlinks (e.g. fixup absolute symlinks that should have been relative). The latter is closer to a ln -s call.
• If you need to rename a file, please consider using rename_path (since 10.5). It behaves mostly like mv -f but requires dest to be a (non-existing) file.
• Have a look at whether on_pkgs_in_parallel() / on_items_in_parallel() would be suitable for enabling parallelization in your tool.
  • The emphasis for these functions is on making parallelization easy to add with minimal code changes. It pre-distributes the items which can lead to unbalanced workloads, where some processes are idle while a few keeps working.

Credits: I would like to thank the following for reporting performance issues, regressions or/and providing patches. The list is in no particular order:

• Helmut Grohne
• Kurt Roeckx
• Gianfranco Costamagna
• Iain Lane
• Sven Joachim
• Adrian Bunk
• Michael Stapelberg

Should I have missed your contribution, please do not hesitate to let me know.
Filed under: Debhelper, Debian

Here's what happened in the Reproducible Builds effort between Sunday July 9 and Saturday July 15 2017: Packages reviewed and fixed, and bugs filed

• Adrian Bunk:
  • #867771 filed against python-trollius.
  • #867773 filed against relatorio.
  • #867781 filed against gcin.
  • #867890 filed against apache-directory-jdbm.
  • #867900 filed against yapsy.
  • #867906 filed against wcsaxes.
• Chris Lamb:
  • #867753 filed against grunt (forwarded upstream).
  • #867848 filed against gconf (forwarded upstream).
  • #868133 filed against grep.
  • #868321 filed against node-marked-man (forwarded-upstream).
• Bernhard M. Wiedemann:
  • chromium
  • chromium
  • kubernetes
  • qtscriptgenerator
  • crash merged
  • ghc-rpm-macros merged
  • samba
  • htmldoc
  • efl
  • webkitgtk3 merged
  • fence-agents
  • fio merged

Reviews of unreproducible packages 13 package reviews have been added, 12 have been updated and 19 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added:

• build_path_captured_by_valac
• timestamps_in_javascript_generated_by_node_grunt_banner

3 issue types have been updated:

• Add fix for timestamps_embedded_in_manpages_by_node_marked_man toolchain issue.
• Add fix for timestamps_in_javascript_generated_by_node_grunt_banner toolchain issue.
• timestamps_in_javascript_generated_by_node_grunt_banner is actually in src:grunt

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (47)

diffoscope development Version 84 was uploaded to unstable by Mattia Rizzolo. It included contributions already reported from the previous weeks, as well as new ones:

• Ximin Luo:
  • Attempt to fix fsimage test on Jenkins
  • Use a tempdir rather than ./cache for guestfs cache
  • Add more debugging output to test_fsimage

After the release, development continued in git with contributions from:

• Mattia Rizzolo:
  • RequiredToolNotFound.get_package(): just call the new get_package_provider()
  • Add a get_package_provider() function, returning the package name that best matches the system
  • Move from deprecated platform.linux_distribution() to the external package distro

strip-nondeterminism development Versions 0.036-1, 0.037-1 and 0.038-1 were uploaded to unstable by Chris Lamb. They included contributions from:

• Niels Thykier:
  • Add missing use statements in handler modules
  • dh_strip_nondeterminism: Assumes tmpdir() exists
  • File::StripNondeterminism:
    • Apply perltidy
    • Lazy load most handlers
    • Lazy load remaining handlers
• Chris Lamb:
  • Add missing File::Temp imports in JAR and PNG handlers. This appears to have been exposed by lazily-loading handlers in #867982.

reprotest development Development continued in git with contributions from:

• Ximin Luo:
  • presets: use newer flag --no-sign for dpkg-buildpackage
  • Document --diffoscope-args= --exclude-directory-metadata and use it in presets
• Mattia Rizzolo:
  • Bump debhelper compat level to 10.
  • Bump Standards-Version to 4.0.0.

buildinfo.debian.net development

• Chris Lamb:
  • Avoid a race condition between check-and-creation of Buildinfo instances.

tests.reproducible-builds.org

• Mattia Rizzolo:
  • Make database backups quicker to restore by avoiding --column-inserts's pg_dump option.
  • Fixup the deployment scripts after the stretch migration.
  • Fixup Apache redirects that were broken after introducing the buster suite
  • Fixup diffoscope jobs that were not always installing the highest possible version of diffoscope
• Holger Levsen:
  • Add a node health check for a too big jenkins.log.

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday July 2 and Saturday July 8 2017: Reproducible work in other projects Ed Maste pointed to a thread on the LLVM developer mailing list about container iteration being the main source of non-determinism in LLVM, together with discussion on how to solve this. Ignoring build path issues, container iteration order was also the main issue with rustc, which was fixed by using a fixed-order hash map for certain compiler structures. (It was unclear from the thread whether LLVM's builds are truly path-independent or rather that they haven't done comparisons between builds run under different paths.) Bugs filed

• Adrian Bunk:
  • #867499 filed against tiptop.
  • #867773 filed against relatorio.
  • #867781 filed against gcin.
• Chris Lamb:
  • #866945 filed against tinymux.
  • #867753 filed against grunt.
  • #867848 filed against gconf.

Patches submitted upstream:

• Bernhard M. Wiedemann:
  • perl-Class-MethodMaker sort hashtable
  • perl-Mouse sort file list
  • thunderbird sort symbol list
  • SOURCE_DATE_EPOCH support:
    • graphviz, merged
    • gnupg, merged in improved variant by upstream
    • janus-gateway, merged
    • libkcapi merged
  • txt2tags fixup on earlier patch, merged

Reviews of unreproducible packages 52 package reviews have been added, 62 have been updated and 20 have been removed in this week, adding to our knowledge about identified issues. No issue types were updated or added this week. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (143)
• Andreas Beckmann (1)
• Dmitry Shachnev (1)
• Lucas Nussbaum (3)
• Niko Tyni (3)
• Scott Kitterman (1)
• Sean Whitton (1)

diffoscope development Development continued in git with contributions from:

• Ximin Luo:
  • Add a PartialString class to help with lazily-loaded output formats such as html-dir.
  • html and html-dir output:
    • add a size-hint to the diff headers and lazy-load buttons
    • add new limit flags and deprecate old ones
  • html-dir output
    • split index pages up if they get too big
    • put css/icon data in separate files to avoid duplication
  • main: warn if loading a diff but also giving diff-calculation flags
  • Test fixes for Python 3.6 and CI environments without imagemagick (#865625).
  • Fix a performance regression (#865660) involving the Wagner-Fischer algorithm for calculating levenshtein distance.

With these changes, we are able to generate a dynamically loaded HTML diff for GCC-6 that can be displayed in a normal web browser. For more details see this mailing list post. Misc. This week's edition was written by Ximin Luo, Bernhard M. Wiedemann and Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday June 25 and Saturday July 1 2017: Upcoming and past events Our next IRC meeting is scheduled for July 6th at 17:00 UTC (agenda). Topics to be discussed include an update on our next Summit, a potential NMU campaign, a press release for buster, branding, etc. Toolchain development and fixes

• James McCoy reviewed and merged Ximin Luo's script debpatch into the devscripts Git repository. This is useful for rebasing our patches onto new versions of Debian packages.

Packages fixed and bugs filed

• Adrian Bunk:
  • #866713 filed against debhelper.
• Chris Lamb:
  • #865994 filed against xabacus.
  • #866164 filed against qmidinet.
  • #866169 filed against singularity-container.
  • #866330 filed against cd-hit.

Ximin Luo uploaded dash, sensible-utils and xz-utils to the deferred uploads queue with a delay of 14 days. (We have had patches for these core packages for over a year now and the original maintainers seem inactive so Debian conventions allow for this.) Patches submitted upstream:

• openmpi
• gtk-doc fixed by sorting directory listings
• samba
• TeX
• nedit
• criu
• tvheadend sort
• tvheadend date
• cfengine

Reviews of unreproducible packages 4 package reviews have been added, 4 have been updated and 35 have been removed in this week, adding to our knowledge about identified issues. One issue types has been updated:

• Add upstream URL for random_order_of_pdf_ids_generated_by_latex

One issue type has been added:

• timestamps_in_manpages_added_by_golang_go_flags

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (68)
• Daniel Schepler (1)
• Michael Hudson-Doyle (1)
• Scott Kitterman (6)

diffoscope development

• Daniel Shahaf:
  • Fix markup in the man page synopsis. Thanks to Niels Thykier for the report. (Closes: #866577)
• Mattia Rizzolo:
  • Bump backport version check in debian/rules
• Ximin Luo:
  • Fix a progressbar failure
  • Put the 400MB "fsimage" cache in a more obvious place
  • Fix CI tests under Python 3.6
  • Add a --exclude-directory-metadata option. (Closes: #866241)
  • Raise warning for getfacl. Remove a redundant try-clause
  • Fix recursive indentation of headers
  • Use a loop rather than recursion
  • In html-dir mode, put css/icon in separate files to avoid duplication
  • diffcontrol UI tweaks
  • Split index pages up if they get too big

tests.reproducible-builds.org

• Vagrant Cascadian working on testing Debian:
  • Upgraded the 27 armhf build machines to stretch.
  • Fix mtu check to only display status when eth0 is present.
• Helmut Grohne worked on testing Debian:
  • Limit diffoscope memory usage to 10GB virtual per process. It currently tends to use 50GB virtual, 36GB resident which is bad for everything else sharing the machine. (This is #865660)
• Alexander Couzens working on testing LEDE:
  • Add multiple architectures / targets.
  • Limit LEDE and OpenWrt jobs to only allow one build at the same time using the jenkins build blocker plugin.
  • Move git log -1 > .html to node document environment().
  • Update TODOs.
• Mattia Rizzolo working on testing Debian:
  • Updated the maintenance script for postgresql-9.6.
  • Restored the postgresql database from backup after Holger accidently purged it.
• Holger Levsen working on:
  • Merging all the above commits.
  • Added a check for (known) Jenkins zombie jobs and report them. (This is an long known problem with jenkins; deleted jobs sometimes come back )
  • Upgraded the remaining amd64 nodes to stretch.
  • Accidentially purged postgres-9.4 from jenkins, so we could test our backups
  • Updated our stretch upgrade TODOs.

Misc. This week's edition was written by Chris Lamb, Ximin Luo, Holger Levsen, Bernhard Wiedemann, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday June 18 and Saturday June 24 2017: Upcoming and Past events Our next IRC meeting is scheduled for the 6th of July at 17:00 UTC with this agenda currently:

1. Introductions
2. Reproducible Builds Summit update
3. NMU campaign for buster
4. Press release: Debian is doing Reproducible Builds for Buster
5. Reproducible Builds Branding & Logo
6. should we become an SPI member
7. Next meeting
8. Any other business

On June 19th, Chris Lamb presented at LinuxCon China 2017 on Reproducible Builds. On June 23rd, Vagrant Cascadian held a Reproducible Builds question and answer session at Open Source Bridge. Reproducible work in other projects LEDE: firmware-utils and mtd-utils/mkfs.jffs2 now honor SOURCE_DATE_EPOCH. Toolchain development and fixes There was discussion on #782654 about packaging bazel for Debian. Dan Kegel wrote a patch to use ar determinitiscally for Homebrew, a package manager for MacOS. Dan Kegel worked on using SOURCE_DATE_EPOCH and other reproduciblity fixes in fpm, a multi plattform package builder. The Fedora Haskell team disabled parallel builds to achieve reproducible builds. Bernhard M. Wiedemann submitted many patches upstream:

• Sorting file lists:
  • python setuptools
  • cpython tarfile/zipfile
  • lyx
  • dpdk
  • perl Alien::SDL
  • gtk-doc, no fix yet
• Use date from SOURCE_DATE_EPOCH:
  • uwsgi
  • recode
  • perl Glib

Packages fixed and bugs filed Patches submitted upstream:

• Bernhard M. Wiedemann
  • samba, sorting file lists
  • open-mpi, merged, sorting file lists
  • virtualbox, incomplete fix, use gzip -n
  • xen/etherboot, xen/mini-os, use gzip -n
  • nautilus-dropbox, drop date
  • samba, use date from SOURCE_DATE_EPOCH.

Other patches filed in Debian:

• Adrian Bunk:
  • #865577 filed against cloud-init.
  • #865813 filed against jumpnbump.
• Chris Lamb:
  • #865554 filed against janus.
  • #865623 filed against cracklib2.
  • #865751 filed against nfstrace.
  • #865753 filed against tigervnc.

Reviews of unreproducible packages 573 package reviews have been added, 154 have been updated and 9 have been removed in this week, adding to our knowledge about identified issues. 1 issue type has been updated:

• random_order_of_pdf_ids_generated_by_latex, added the name of the function that possibly causes this issue.

Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (98)

diffoscope development Version 83 was uploaded to unstable by Chris Lamb. It also moved the previous changes from experimental (to where they were uploaded) to unstable. It included contributions from previous weeks. You can read about these changes in our previous weeks' posts, or view the changelog directly (raw form). We plan to maintain a backport of this and future versions in stretch-backports. Ximin Luo also worked on better html-dir output for very very large diffs such as those for GCC. So far, this includes unreleased work on a PartialString data structure which will form a core part of a new and more intelligent recursive display algorithm. strip-nondeterminism development Versions 0.035-1 was uploaded to unstable from experimental by Chris Lamb. It included contributions from:

• Bernhard M. Wiedemann
  • Add CPIO handler and test case.
• Chris Lamb
  • Packaging improvements.

Later in the week Mattia Rizzolo uploaded 0.035-2 with some improvements to the autopkgtest and to the general packaging. We currently don't plan to maintain a backport in stretch-backports like we did for jessie-backports. Please speak up if you think otherwise. reproducible-website development

• Chris Lamb:
  • Add OpenEmbedded to projects page after a discussion at LinuxCon China.
  • Update some metadata for existing talks.
  • Add 13 missing talks.

tests.reproducible-builds.org

• Alexander 'lynxis' Couzens
  • LEDE: do a quick sha256sum before calling diffoscope. The LEDE build consists of 1000 packages, using diffoscope to detect whether two packages are identical takes 3 seconds in average, while calling sha256sum on those small packages takes less than a second, so this reduces the runtime from 3h to 2h (roughly). For Debian package builds this is neglectable, as each build takes several minutes anyway, thus adding 3 seconds to each build doesn't matter much.
  • LEDE/OpenWrt: move toolchain.html creation to remote node, as this is were the toolchain is build.
  • LEDE: remove debugging output for images.
  • LEDE: fixup HTML creation for toolchain, build path, downloaded software and GIT commit used.
• Mattia 'mapreri' Rizzolo:
  • Debian: introduce Buster.
  • Debian: explain how to migrate from squid3 (in jessie) to squid (in stretch).
• Holger 'h01ger' Levsen:
  • Debian:
    • Add jenkins jobs to create schroots and configure pbuilder for Buster.
    • Add Buster to README/about jenkins.d.n.
    • Teach jessie and ubuntu 16.04 systems how to debootstrap Buster.
    • Only update indexes and pkg_sets every 30min as the jobs almost run for 15 min now that we test four suites (compared to three before).
    • Create HTML dashboard, live status and dd-list pages less often.
    • (Almost) stop scheduling old packages in stretch, new versions will still be scheduled and tested as usual.
    • Increase scheduling limits, especially for untested, new and depwait packages.
    • Replace Stretch with Buster in the repository comparison page.
    • Only keep build_service logs for a day, not three.
    • Add check for hanging mounts to node health checks.
    • Add check for haveged to node health checks.
    • Disable ntp.service on hosts running in the future, needed on stretch.
    • Install amd64 kernels on all i386 systems. There is a performance issue with i386 kernels, for which a bug should be filed. Installing the amd64 kernel is a sufficient workaround, but it breaks our 32/64 bit kernel variation on i386.
  • LEDE, OpenWrt: Fix up links and split TODO list.
  • Upgrade i386 systems (used for Debian) and pb3+4-amd64 (used for coreboot, LEDE, OpenWrt, NetBSD, Fedora and Arch Linux tests) to Stretch
  • jenkins: use java 8 as required by jenkins >= 2.60.1

Misc. This week's edition was written by Ximin Luo, Holger Levsen, Bernhard M. Wiedemann, Mattia Rizzolo, Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday June 11 and Saturday June 17 2017: Upcoming events

• On June 19th, Chris Lamb presented at LinuxCon China 2017 on Reproducible Builds.
• h01ger created a poll for a date for the next Reproducible Builds summit, please vote if you are interested in attending.
• Our next IRC meeting will be on the 6th of July at 17:00 UTC with the following agenda.

Upstream patches and bugs filed

• Bernhard M. Wiedemann:
  • gnuradio + volk
  • pymol
  • distorm
  • qtscriptgenerator
  • cpython
  • x3270 x3270
  • sphinx
  • obs-service-tar_scm
  • osc
  • matplotlib merged
  • pyparted merged
  • bjoern merged

Reviews of unreproducible packages 1 package review has been added, 19 have been updated and 2 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (1)
• Edmund Grimley Evans (1)

diffoscope development

• Chris Lamb:
  • Document the 'h' variable in our raw_feeder
  • Split diffoscope.difference into feeders
  • Tidy diffoscope/difference.py
• Ximin Luo:
  • Add a PartialString class
  • More ydiff/linediff from diffoscope. difference => diff to group unified_diff related things together
  • difference: has_children -> has_visible_children, and take into account comments
  • Add various traverse_ methods to Difference
  • Move side-by-side and linediff algorithms to difference.py
  • Refactor html-dir presenter to be a class instance, avoiding global state
  • html-dir: show/hide diff comments, which can be very large

tests.reproducible-builds.org As you might have noticed, Debian stretch was released last week. Since then, Mattia and Holger renamed our testing suite to stretch and added a buster suite so that we keep our historic results for stretch visible and can continue our development work as usual. In this sense, happy hacking on buster; may it become the best Debian release ever and hopefully the first reproducible one!

• Vagrant Cascadian:
  • Proposed reducing the number of concurrent builds each armhf machine runs, to reduce out-of-memory errors, crashes and false positives with FTBFS on armhf, reducing the number of armhf build workers from 70 to 51.
• Valerie Young: Add highlighting in navigation for the new nodes health pages.
• Mattia Rizzolo:
  • Do not dump database ACL in the backups.
  • Deduplicate SSLCertificateFile directive into the common-directives-ssl macro
  • Apache: t.r-b.o: redirect /testing/ to /stretch/
  • db: s/testing/stretch/g
  • Start adding code to test buster...
• Holger Levsen:
  • Update README.infrastructure to explain who has root access where.
  • reproducible_nodes_info.sh: correctly recognize zero builds per day.
  • Add build nodes health overview page, then split it in three: health overview, daily munin graphs and weekly munin graphs.
  • reproducible_worker.sh: improve handling of systemctl timeouts.
  • reproducible_build_service: sleep less and thus restart failed workers sooner.
  • Replace ftp.(de uk us).debian.org with deb.debian.org everywhere.
  • Performance page: also show local problems with _build_service.sh (which are autofixed after a maximum of 133.7 minutes).
  • Rename nodes_info job to html_nodes_info.
  • Add new node health check jobs, split off from maintenance jobs, run every 15 minutes.
    • Add two new checks: 1. for correct future (2019 is incorrect atm, and we sometimes got that). 2.) for writeable /tmp (sometimes happens on borked armhf nodes).
  • Add jobs for testing buster.
  • s/testing/stretch/g in all the code.
  • Finish the code to deal with buster.
  • Teach jessie and Ubuntu 16.04 how to debootstrap buster.

Axel Beckert is currently in the process of setting up eight LeMaker HiKey960 boards. These boards were sponsored by Hewlett Packard Enterprise and will be hosted by the SOSETH students association at ETH Zurich. Thanks to everyone involved here and also thanks to Martin Michlmayr and Steve Geary who initiated getting these boards to us. Misc. This week's edition was written by Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Here's what happened in the Reproducible Builds effort between Sunday June 4 and Saturday June 10 2017: Past and upcoming events On June 10th, Chris Lamb presented at the Hong Kong Open Source Conference 2017 on reproducible builds. Patches and bugs filed

• Chris Lamb: #864082 filed against fontconfig, forwarded upstream

Reviews of unreproducible packages 7 package reviews have been added, 10 have been updated and 14 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (4)
• Chris Lamb (1)
• Christoph Biedl (1)
• Niko Tyni (1)

Two FTBFS issues of LEDE (exposed in our setup) were found and were fixed:

• build: ensure that flock is available for make download (Felix Fietkau)
• include/toplevel: set env GIT_ASKPASS=/bin/true (Alexander 'lynxis' Couzens)

diffoscope development

• Chris Lamb: Some code style improvements

tests.reproducible-builds.org: Alexander 'lynxis' Couzens made some changes for testing LEDE and OpenWrt:

• Build tar before downloading everything: On system without tar --sort=name we need to compile tar before downloading everything
• Set CONFIG_AUTOREMOVE to reduce required space
• Create a workaround for signing keys: LEDE signs the release with a signing key, but generates the signing key if it's not present. To have a reproducible release we need to take care of signing keys.
• openwrt_get_banner(): use staging_dir instead of build_dir because the former is persistent among the two builds.
• Don't build all packages to improve development speed for now.
• Only build one board instead of all boards. Reducing the build time improves developing speed. Once the image is reproducible we will enable more boards.
• Disable node_cleanup_tmpdirs

Hans-Christoph Steiner, for testing F-Droid:

• Do full git reset/clean like Jenkins does
• hard code WORKSPACE dir names, as WORKSPACE cannot be generated from $0 as it's a temporary name.

Daniel Shahaf, for testing Debian:

• Remote scheduler:
  • English fix to error message.
  • Allow multiple architectures in one invocation.
  • Refactor: Break out a helper function. Rename variable to disambiguate with scheduling_args.message.
• Include timestamps in logs
• Set timestamps to second resolution (was millisecond by default).

Holger 'h01ger' Levsen, for testing Debian:

• Improvements to the breakages page:
  • List broken packages and diffoscope problems first, and t.r-b.o problems last.
  • Reword, drop 'caused by'.
• Add niceness to our list of variations, running with niceness of 11 for the first build and niceness of 10 for the second one. Thanks to Vagrant for the idea.
• Automatic scheduler:
  • Reschedule after 12h packages that failed with error 404
  • Run scheduler every 3h instead of every 6h
• Add basic README about the infrastructure and merge Vagrants notes about his console host.

Misc. This week's edition was written by Ximin Luo, Chris Lamb and Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.